Data Distributions

       

• Directives:

  !HPF$ PROCESSORS processor_array_name(dim1,dim2,..,dimN)
  !HPF$ ALIGN array WITH target_array
  !HPF$ DISTRIBUTE list_of_arrays [ONTO processor_array_name]
  

  For example:

  !HPF$ PROCESSORS line(12)
  !HPF$ PROCESSORS matrix_1(3,4)
  !HPF$ PROCESSORS matrix_2(4,3)
  

• The line arrangement is usually most efficient on systems with slow communications, e.g., ATM or FDDI farms
• Alignment:
  
   

  Figure 3.1: Visualisation of alignment (from Foster)

  
  
  
  • The ALIGN directive must precede the DISTRIBUTE directive
  • Another example of alignment
    
    

     

    Figure 3.2: Example of alignment from Marshall

    real, dimension(5) :: d real, dimension(10) :: e !HPF$ ALIGN d(:) WITH e(1::2) !HPF$ DISTRIBUTE e(BLOCK) d = d + e(::2)

    
    
    The alignment guarantees that the additions are local everywhere.
• Distribution:
  
   

  Figure 3.3: Possible distributions (from Foster)

  
  
     

  *	No distribution
  BLOCK(n)	Block distribution, default n = N/P
  CYCLIC(n)	Cyclic distribution, default n = 1

• Examples:

  1.

  program hpf_laplace
  !HPF$ PROCESSOR line(4)
     real X(100,100), New(100,100)
  !HPF$ ALIGN New(:,:) WITH X(:,:)
  !HPF$ DISTRIBUTE X(BLOCK,*) ONTO line
     New(2:99,2:99) = (X(1:98,2:99) + X(3:100,2:99) &
                     + X(2:99,1:98) + X(2:99,3:100))/4
     diffmax = MAXVAL(ABS(New-X))
  end
  

  2.

  program hpf_pairwise_interactions
  !HPF$ PROCESSORS POOL_2(12)
     integer i,n
     parameter (n = 768) ! 12 * 64
     real R1(3,n), F(3,n), R2(3,n)
     integer i
  !HPF$ ALIGN F(:,:) WITH R1(:,:)
  !HPF$ ALIGN R2(:,:) with R1(:,:)
  !HPF$ DISTRIBUTE R1(*,BLOCK) ONTO POOL_2
     F = 0.0
     R2 = R1
     do i = 1,n-1
        R2 = CSHIFT(R2,1,2)
        F = F + Gravity(R1,R2)
     enddo
  end
  

  Watch for a bug in Foster: he rotates columns, i.e., the $\vec{r}$ vectors, instead of rows, i.e., particles.

• Distribution of Allocatables:   Distribution directives take effect at allocation:

  REAL, ALLOCATABLE, DIMENSION(:,:) :: A
  INTEGER                           :: ierr
  !HPF$ PROCESSORS, DIMENSION(10,10)    :: P
  !HPF$ DISTRIBUTE (BLOCK,CYCLIC)       :: A
        ...
       ALLOCATE(A(100,20),stat=ierr)
        ...
       DEALLOCATE(A)
      END
  

  The array A is automatically distributed when it is allocated with block size of 10 in dimension 1.

• Mapping and Procedures:     In procedures distributions can be either prescriptive (i.e., the same as in main) or descriptive. If prescriptive distributions differ from distributions in the calling program, the passed arrays will be remapped. Even if they don't differ, remapping may still be attempted. On exit original distribution is restored.
  • Example of a Prescriptive Distribution

    SUBROUTINE Subby(A,B,RES)
       IMPLICIT NONE
       REAL, DIMENSION(:,:), INTENT(IN)   :: A, B
       REAL, DIMENSION(:,:), INTENT(OUT)  :: RES
    !HPF$ PROCESSORS, DIMENSION(2,2)      :: P
    !HPF$ TEMPLATE, DIMENSION(4,6)        :: T
    !HPF$ ALIGN (:,:) WITH T(:,:)         :: A, B, RES
    !HPF$ DISTRIBUTE (BLOCK,BLOCK) ONTO P :: T
       ...
    END SUBROUTINE Subby
    

      Observe the use of the TEMPLATE.
  • Example of a Descriptive Distribution

    SUBROUTINE Subby(A,B,RES)
       IMPLICIT NONE
       REAL, DIMENSION(:,:), INTENT(IN)     :: A, B
       REAL, DIMENSION(:,:), INTENT(OUT)    :: RES
    !HPF$ PROCESSORS, DIMENSION(2,2)        :: P
    !HPF$ TEMPLATE, DIMENSION(4,6)          :: T
    !HPF$ DISTRIBUTE *(BLOCK,BLOCK) ONTO *P :: T
    !HPF$ ALIGN (:,:) WITH *T(:,:)          :: A, B, RES
       ...
    END SUBROUTINE Subby
    

    Here we assert that the arrays A, B, and RES are distributed as shown.

  • Dummy Distributions
    • DISTRIBUTE (CYCLIC) ONTO P :: A
      • distribute A cyclically onto P
    • DISTRIBUTE *(CYCLIC) ONTO P :: A
      • A already has cyclic distribution but may not be distributed over P.
    • DISTRIBUTE (CYCLIC) ONTO *P :: A
      • A is distributed over P but may not have CYCLIC distribution.
    • DISTRIBUTE *(CYCLIC) ONTO *P :: A
      • A already has cyclic distribution over P

  • The INHERIT Directive:   In fully blown HPF, there is the

    !HPF$ INHERIT   A
    

    directive. The currently available subset HPF does not support this directive though, because it is too difficult to generate code that can handle multiple distributions.

• System Inquiry Functions:      

  PROCESSORS_SHAPE()	$\rightarrow$	/4,8/
  NUMBER_OF_PROCESSORS()	$\rightarrow$	32
  NUMBER_OF_PROCESSORS(1)	$\rightarrow$	4
  NUMBER_OF_PROCESSORS(2)	$\rightarrow$	8

  System inquiry functions can be also included in HPF directives, e.g.,

  !HPF$ PROCESSORS P(NUMBER_OF_PROCESSORS())
     INTEGER Q(SIZE(PROCESSORS_SHAPE()))