Supercomputing & Undergraduate Education at the San Diego Supercomputer Center

Poster Presentation at Supercomputing '92

Dr. Kris Stewart

Associate Professor, Dept. of Mathematical Sciences
Senior Fellow, San Diego Supercomputer Center
San Diego State University
San Diego, CA 92182-0314
stewart@cs.sdsu.edu

Dec. 2, 1992

SUE was funded primarily by the Division of Advanced Scientific Com­puting at the National Science Foundation. Additional funding was provided by The Cray Research Foundation.

A Background on the grant

The San Diego Supercomputer Center (SDSC) has taken a proactive role in disseminating information to make supercomputing accessible to a much wider audience. In particular, SDSC has targeted instructors at undergraduate institutions to introduce their students to supercom­puters and their use. As part of this effort, SDSC established Super­computing and Undergraduate Education (SUE). This program enhances the supercomputing expertise of faculty and helps them incorporate supercomputing topics into their curricula and departmental majors.

A major component of this program is a one-week residential summer workshop at SDSC for faculty from primarily undergraduate institu­tions throughout the U.S. Lecture materials from both the 1991 and 1992 workshops can be obtained via anonymous FTP over the Internet (See handout for details).
Another component of this program is an annual course taught at San Diego State University (SDSU), in which undergraduate students learn about supercomputers and their use. This course is described in the SDSU catalog as follows:

CS 575 - Supercomputing for the Sciences

SUE's 1991 and 1992 faculty workshops at SDSC and the undergraduate curricula at SDSU focused on the following topics in supercomputing.

The workshop faculty represented two groups: those interested in learning discipline-specific applications packages and those inter­ested in using software tools to facilitate programming. To accommo­date both groups, we presented an overview of available resources (applications packages and program optimization techniques) and encouraged faculty to seek further information independently. We identified the technical people they could contact for further infor­mation on their particular interests.

Similarly, the required programming background for the course rein­forced the traditional separation between computer science and sci­ence/engineering students by encouraging the former to take the course. In both cases, the participants interacted well as they real­ized the benefits of working with others with different strengths.

2 CRAY architecture

The course used the text Computer Architecture: A Quantitative Approach, by Hennessy and Patterson (published by Morgan Kaufmann). The sections we used from this text proved invaluable.

We tried to gain an appreciation for the sources of the Cray Y-MP's power and understand the subtleties of its design as they impact the way a programmer should approach a problem. Therefore, we carefully avoided many of the sections of the text that did not directly relate to the Cray Y-MP. (The course notes available via anonymous FTP can provide a guide for other instructors interested in using this text for a similar purpose.) This text was supplemented with documents by Cray Research, Inc.; particularly the following titles:

TR-OPT (Rev. D) cf77 & scc Features and Optimization. An excellent, Cray-specific training report covering the Fortran (cf77) and C (scc) compilers. Provides code examples, diagrams, and explanations of crucial vectorization topics (and the conditions that inhibit the compiler), memory organization, performance tools, common optimiza­tion techniques, and much more.

TR-YSAAP Cray Y-MP System Architecture for Applications Programmers. Covers material at the assembler level in more detail. Works well with the models developed in Hennessy and Patterson (unfortunately this document is no longer available)

3 Architecture of parallel supercomputers
(Intel iPSC/860 and nCUBE 2)

This was not covered in the undergraduate course at SDSU, but was covered at the faculty workshop by consultants from Intel and nCUBE, who gave introductory lectures on the parallel architectures.

Both the workshop and the course covered the resources

a. Accessing sources of information

b. Communicating with peers (via e-mail)

c. Using FTP to transfer programs between machines (a
crude look at heterogeneous computing)

5 Accounting

At many universities, accounts issued to students in computer courses are limited only by the amount of available file space. Therefore, the concept of monitoring CPU usage can be new. The students in the SDSU class were given programming projects on a mainframe at SDSU and were introduced to the concept of accounting using crude UNIX timing tools (e.g., dtime) before they were moved to the Cray.

When they ran programs on the Cray, students and faculty had a fixed amount of CPU time to work with. Students were reminded that they must complete their course projects without exceeding their CPU allo­cations. The more sophisticated timing and resource monitoring tools were used to ensure that the student projects did not use up more time than they were allocated for the semester.

6 Computer ethics/responsibility

Students in the course chose between two computer-related scenarios and wrote a one-page essay on a scenario, discussing the behaviors of the individuals involved. The essays were graded on a pass/fail basis (only essays that showed a lack of thought or effort "failed"). The goal, for the instructor, was to gain a better under­standing of the students' attitudes concerning computer-related issues.

Dr. Dan Sulzbach, Executive Director of SDSC, also gave a week of lectures on computer ethics, which stimulated some very interesting class discussions.

Dr. Sulzbach began his talk by stating:

Computer accounts on the Cray Y-MP were distributed only after the week-long discussions on ethics.

Most of the students had no problem with the computer ethics essay, but they typically had little experience in writing up a science-oriented programming report. Therefore, we assigned a programming project that consisted of the following components:

a. Solve a problem on a campus mainframe and document the mainframe's performance.

b. Solve the same problem on the Cray.

c. Compute an enhancement of the problem on the Cray and document the Cray's performance.

Instructor feedback after reading the document from assignment (a) greatly enhanced the organization and content of the final document in (c).

See Section G SDSU course programming assignments in this handout for more details on the exact assignments. Section G also gives the instructor's clarifications and directions to students to help them specify and document their projects.


C SUE Workshop Overview

This one-week workshop overviewed the many resources available at the San Diego Supercomputer Center. In general, he mornings involved pre­sentations by consultants from the following institutions:

San Diego Supercomputer Center (SDSC)
Cray Research, Incorporated (CRI)
Intel Supercomputer Systems
nCUBE Corporation

In the afternoons, software demonstrations or open labs were orga­nized. Due to the diversity of backgrounds of the workshop partici­pants, we covered a very wide variety of information at a basic level. Participants were encouraged to discuss the information in more detail with the consultants in the afternoon laboratory ses­sions. We also provided information on resources available over the Internet to enhance curriculum development in supercomputing.

Technical support was provided by

Cray Research, Incorporated
Intel Corporation
nCUBE Corporation
D SUE Workshop Schedule
MONDAY

am Welcome

pm Afternoon lab session

TUESDAY

TR-OPT is an extensive Cray software training workbook which is typically covered at a more leisurely pace over a one-week period. Etan will try to highlight crucial portions and will then be avail­able for individual discussions this afternoon and tomorrow.

Etan will be available all afternoon to answer any Cray-specific questions.

WEDNESDAY

am Cray Application Packages (SDSC Consultants)
Biology (Jack Rogers)
Chemistry (Jerry Greenberg)
CFD (Rich Charles)
Math (Bob Leary)

pm "Teaching Chemistry" (Rozeanne Steckler)

THURSDAY

pm Scalable Version of Wave Equation (Carl Scarbnick, SDSC)

am Discussions

Most of the files from the SUE workshop can be accessed via anonymous ftp. To access them, FTP to the host rohan.sdsu.edu then retrieve from the directories /pub/sdscinfo/SUE-notes, /pub/sdscinfo/SDSC-info-files or /pub/sdscinfo/Supercomputing-Course-Notes.

A short description of the individual files is contained at the end of this handout.

F Lecture materials and readings

The sections from the following chapters of Computer Architecture: A Quantitative Approach (Hennessy & Patterson) were covered in the first nine weeks of lectures in the course. See the lecture notes available via anonymous FTP for more details on the sections covered.

Chap. 1 Fundamentals of Computer Design
Chap. 2 Performance and Cost
Chap. 3 Instruction Set Design: Alternatives and Principles
Chap. 4 Instruction Set Examples and Measurements of Use
Chap. 5 Basic Processor Implementation Techniques
Chap. 6 Pipelining
Chap. 7 Vector Processors
Chap. 8 Memory-Hierarchy
The goal was to understand the following advanced topics:

Pipelined (segmented) functional units

As the lecture material was covered, students worked on their first two programming assignments. The goal was to develop a feeling for timings on the SDSU mainframe compared to accuracy of approximation schemes. Students were asked to assess how much it costs (measured in CPU time for now) to get a good answer (measured by true error). Since most students had little numerical analysis background, Dr. Stewart provided the original code for the "First Program" problem, and the students were instructed to insert the appropriate timing calls.

First Program 1991 Course

Run and time a Fortran code that solves a two-point boundary value problem

The dimension of the approximation should be varied and you should time the separate pieces of the solution process. You should docu­ment the performance of the SDSU mainframe on this problem and dis­cuss the sensitivity in accuracy and timing

NOTE: This proved to be a somewhat confusing problem for students new to numerical analysis. There are too many sources of error. As the finite difference grid is refined, the approximation is more accurate. But the linear system that is solved becomes more poorly conditioned, thereby introducing errors.
First Program 1992 Course

Consider the linear system A x = b, where the N by N matrix A is given by

a_ij = 1/(i+j-1) (the notorious Hilbert matrix)

The right-hand side, b, will be chosen so that the true solution, x, will be all 1s. Therefore,

b_j = S a_ij

You should solve this linear system for various values of N and observe the error incurred (by computing true error, since we know the true solution should have X = 1s) and the performance (measured by the elapsed CPU time).

Science-Oriented Program to be run on SDSU Mainframe and subsequently on the CRAY

The main project was crucial to this course. Students were going to use the Cray to run this project, and the instructor did not want them to squander Cray resources before they became familiar with their problems. Students were allowed to pick from a selec­tion of problems provided by the instructor, or they could solve a scientific problem from their particular backgrounds or work environments.

Good sources for problem statements were:

"Computing Applications to Differential Equations: Modelling in the Physical and Social Sciences," by J.M.A. Danby; Reston Publishing

"Numerical Methods and Software" by Kahaner, Moler and Nash; Pren­tice-Hall Publishers.

"Computational Physics" by Koonin and Meredith; Addison Wesley Pub­lishing Company.


The MAIN PROJECT assignment:

a) Get your "science" program running on the SDSU mainframe.

a) Your write-up should have a self-contained statement of the problem. The reader should not have to read your code to find out what equa­tions you are working with, or what the specific problem is that you are solving.

b) Give a complete reference to where the problem came from.

c) Define your measure of work so that comparisons can be made when you run on the Cray. You can't talk about "faster" or "better" with­out a specific measure of performance.

d) Discuss what conclusions are drawn from the problem itself. What is the "science" story revealed by the original problem? Why was this problem solved.

e) You should carefully organize the results. A summary of pertinent results for both the "science" of the problem and the "performance" of the program should be presented. Optionally, include an appendix for more detailed results.

Midterm Exam for the 1992 course

This midterm focused on compiler terminology and related concepts from the Hennessy/Patterson text with the Cray document TR-OPT, including the following :

Jamming Vectorizing Loops

The midterm from the 1991 course asked students to write and time DLXV assembly code (developed in great detail in the Hennessy and Patterson text with numerous examples) for the translation of For­tran to perform the matrix/vector multiply, Ax = b, in two differ­ent manners.

Row-oriented manner we learn first:

j*,b> = x_j , j=1,...,N

Column-oriented manner more suitable for vector processors:

b A + ... + b A = x
1 *1 N *N

This is a demanding problem, but most students gain a deep under­standing of the Cray's vector processor structure.

HPCC and K-8 Education

Jayne Keller, SDSC Education Coordinator

Integrating high-performance computing and communications (HPCC) into the curriculum of primary and secondary schools is critical to the development of the technicians, scientists, and engineers of the future. SDSC offers the following activities to address this need: Supercomputer Center field trip, HPCC half-day in-service workshop, the SDSC road show, and a technology checklist.

Computational Chemistry

Rozeanne Steckler, SDSC Manager, Applications R & D
SDSU Adjunct Professor, Chemistry

The SDSU course Chemistry 596: Chemistry on Supercomputers is designed as an overview of modern computational chemistry with an emphasis on learning to use the major chemistry software packages. This course is not designed as an introduction to theoretical chem­istry, but rather a course to introduce experimental chemists to the computational tools available and how to use them in an informed manner. Many aspects of computational chemistry will be introduced with each topic presented in coordinated lectures and labs.

Computer Graphics

Michael J. Bailey, SDSC Manager of Scientific Visualization

The UCSD course AMES 293 Advanced Computer Graphics for Engineers and Scientists is targeted towards students in engineering or science majors who are interested in applying advanced visualiza­tion techniques to solving scientific problems. It is not oriented towards any one major in particular, but is instead directed towards science in general. Students in this course will learn techniques that will allow them to develop and use scientific graphics programs effectively.

Research Experience for Undergraduates at the San Diego Supercomputer Center

Hassan Aref, SDSC Chief Scientist, and

Students work on research projects in fields of interest within the disciplines that make up computational science. Supervisors are faculty at the student's home institution and SDSC staff. Included are workshops on high-performance computing and special lectures on such topics as parallel computing, graphics and scientific visual­ization, and numerical analysis. Some students, already engaged in computational research with a faculty member, select a topic within that project. Others, with an interest in a certain area, use the REU program to take the first steps. The objective is to give each student a taste of research in computational science, albeit within a condensed time frame. All students are given access to appropriate computer resources at SDSC.

CERFnet

The California Education and Research Federation Network (CERFnet) provides a connection to the world via Internet giving access to hundreds of databases and over a million users worldwide. Its goal is to promote collaboration among scientists, engineers, and educa­tors in commercial, government, and academic sectors.

CERFnet provides a 24-hour hotline, continuous network monitoring and management, an expert staff, and maintenance support. Begun with the support from the National Science Foundation, CERFnet is a project of General Atomics, a San Diego-based research and develop­ment company.

Reuben H. Fleet Space Theater & Science Center: Project Oasis

Joseph Deken, Senior Fellow, SDSC
Senior Scientist, RHF

The Reuben H. Fleet Space Theater and Science Center is one of the most highly respected informal science education centers in the nation. Located in Balboa Park's cultural complex, it houses the world's first OMNIMAX theater, more than 60 "hands-on" science exhibits which encourage visitor participation, as well as a multi­media planetarium show.

In July of 1992, SDSC and the Reuben Fleet Center launched a formal collaboration called Project Oasis. The focus of this collaboration is twofold:
1 To develop interactive exhibits and educational programs about high-performance computing and communications for the general public.
2 To develop new technology for interactive exhibits and educa­tional programs using leading edge computing and communications systems, especially computer networking and visualization.

As part of Project Oasis activities, two SDSC consultants gave lec­tures at the Reuben H. Fleet Space Theater & Science Center recently:

"Computer Visualization I: The Solar System"
by Dave Nadeau, Visualization Specialist at SDSC

"Computer Visualization II: The Antarctic Seafloor"

Kris Stewart, Assoc. Prof., SDSU (stewart@cs.sdsu.edu)

This file (readme.SUE) presents a description of the files distributed at the 1991 Supercomputer and Undergraduate Education (SUE) Workshop. These files are in /pub/sdscinfo/SUE-notes anonymous FTP from rohan.sdsu.edu

Access and use of documents statement (accesuse.asc)
Schedule of events for 1992 SUE workshop (agenda.92)
Brief overview of SDSC accounting and SUE resources
(accounti.sds)
Obtaining Cray documents (cost) (cray-man.cst)
Overview of Dr. Lloyd Fosdick's HPSC program from U. Colorado Boulder. This is a terrific program in undergraduate High Performance Scientific Computing. (hpsc-ks.rme)
Intuitive introduction to ODEs, Euler's method, SAXPY and their
connection with vector operations. This fits well with Chapter 7 from Hennessy and Patterson. (my-chap7.asc)
Kay A. Robbins and Steven Robbins wrote a book that serves as a good teaching tool for understanding the details of the Cray at the assembler level. The Cray X-MP/Model 24: A Case Study in Pipelined Architecture and Vector Processing was published by Springer-Verlag in 1987. (xmpsim1.asc)

1. Organization of CS 575 Supercomputing for the Sciences, taught at San Diego State University, Spring 1991 and 1992, by Kris Stewart

a. Outline of course (575ovrvw.s92)
b. Initial handout to students (575init.asc)
c. List of student programming projects - most final reports and source code are available as tar files (575proj.asc)
d. A road-map through the lecture notes and how they were used in the CS 575 course (readme.575)

2. Actual course notes for CS 575 based on the text, Computer Architecture: A Quantitative Approach by John Hennessy and David Patterson, Morgan Kaufmann Publisher, 1990, coupled with handouts from Cray Research Incorporated from two documents, TR-OPT and TR-YSAAP. The files associated with the Patterson and Hennessy text are named ph-something.asc, those associated with Cray documents are name cray-something.asc. There are many files - see readme.575 in the anonymous ftp directory /pub/sdscinfo/Supercomputing-Lecture-Notes

Of particular interest are the files tr-opt3.asc and tr-opt7.asc

3. Computer Ethics and Responsibility Section. It was felt that before students were given access to the Cray Y-MP it was essential to have an explicit discussion of "ethics" and "responsibility" coupled with a written essay assignment.

a. Computer ethics assignment (four scenarios)
(ethicasg.asc)

b. Handout of the lecture given by Dan Sulzbach (ethic-l2.asc)

4. Using the Cray - note students will have spent 6 weeks programming on the SDSU mainframe in Unix prior to moving to the Cray. (readme.3rd)

b. Man pages for Cray Fortran (cf77, fpp, fmp, cft77) compiling environment (crayacce.asc)

c. Location of sample codes and how to create sample run of Fortran to get listing, marked loops and diagnostics
(cf77 -ZV -Wf"-emx") (crayacce.asc)

d. Man pages for cc and cl for C compiler with listing

e. As in c) above for C to get listing (cl) and diagnostics

I recommend that instructors take extra time to explain reslist (a relatively expensive command which gives students information on their remaining resources), ja (an inexpensive Unix system call with various parameters) and the NQS batch system. You are charged double for all interactive jobs on the Cray at SDSC. Students usually are not familiar with using Batch Queues, which can reduce the charges on a job to 0.5 times actual use (therefore a four-fold decrease over interactive user). Students need to become experienced with these queues to effectively use their finite amount of Cray time.

SDSC Documents (note new users of the SDSC Cray will receive the SDSC User Guide. You can obtain additional copies of the User Guide through the doc processor on the Y-MP. Type doc. The file you want is usrguide. This is a very long file, so I would not recommend getting the whole thing. The individual chapters of the User Guide are available as separate files, e.g. ugoptim, ugtools, ugunicos, etc). Other files available from the doc processor (and in the directory /pub/sdscinfo/SDSC-info-files anonymous FTP from rohan.sdsu.edu):

f. Introduction to UNICOS (unicos)

g. EZFortran, EZC, EZDebug (ezfortrn, ezc, ezdebug)

h. EZStorage (DTI, data tree documentation) (ezstorag)

i. EZMath (math libraries available) (ezmath)

j. EZShell (for those brave Unix hackers who want to customize

k. Optimization - an excellent document, oriented towards

Kris Stewart, Assoc. Prof., SDSU (stewart@cs.sdsu.edu)

This file (readme.2nd) presents a detailed overview of the actual lectures of the CS 575 course. These files are in /pub/sdscinfo/Supercomputing-Course-Notes anonymous FTP from rohan.sdsu.edu

The files fall into three classiciations:

a) course information and additional examples from the instructor

b) those related directly to the Hennessy & Patterson text describing which sections/topics/concepts were used from each chapter

c) xeroxed copied of pages from the Cray documents TR-OPT

ADD R1,R2,R3

This will be important when Chapter 6 discusses pipelining. (add the handout cray-con.asc here)

ph-over6.asc Chapters 6 and 7 are the main goal of the course lecture material. The concepts of:
pipeline operation of segmented scalar computational
units
vector registers
segmented vector functional units and chained pipelines
vector stride, memory bank conflicts and stalls in the
pipe
strip mining vector loops
are covered in these two chapters. Our recurring example will be the saxpy. We first examine Chapter 6 and its sections:

6.1 What is pipelining?

Important homework problems: 6-11 to 6-19 on saxpy

7.7 Putting it all together: Evaluating the performance of vector processors

Kris Stewart, Assoc. Prof., SDSU (stewart@cs.sdsu.edu)

This file (readme.3rd) provides detail pertinent to actually running codes on the Cray Y-MP at SDSC. These files are in /pub/sdscinfo/Supercomputing-Course-Notes anonymous FTP from rohan.sdsu.edu

Students received their own copy of SDSC User's Guide. This is an excellent document written by the SDSC consultants giving a thorough introduction to the Cray, its programming tools, applications packages and more. Given human nature, the User's Guide's thoroughness tends to dissuade users from just sitting down and reading it cover to cover. I therefore e-mailed students a copy of

sdsc-gid.asc A Road Map for the SDSC User's Guide highlighting portions students should pay particular attention to.

crayacce.asc This gives information which instructors could obtain and then make available to their students on their home machines. Includes especially useful man pages, the location on the Cray Y-MP of some sample Fortran and C codes used in TR-OPT. Also gives my recommended calling options for cf77 and cc/cl for generating informative examples and listings.

Students were given the location of crayfopt.asc and cray-exc.asc on our home machine since they are very long. They contain detailed information that students could examine to familiarize themselves with the SDSC Cray tools without using up their own CPU allocation.

Since students will have a finite amount of CPU time on the Cray, I tried to provide a lot of information that most users would obtain in initial explorations on a new machine. I wanted to avoid having 30 separate students repeat the same explorations. Therefore, students were e-mailed a copy of

I think most people will find the initcray.asc file informative, though parts of it are specific to accessing the SDSC Cray from facilities at SDSU. It has hints on Cray usage. It presents a log of an actual session, so that you can see how the logon process proceeds. The Cray news file is read. The doc processor is used to find the file optimiz which is a very informative SDSC Fortran document). This file is copied to a local file (since we see that it is approximately 100 pages long) so that you can edit it, or download it to another machine, or whatever.