This paper proposes a novel texture modeling method based on the pattern spectrum and optimization algorithm. The pattern spectrum is a mathematical morphological method to describe the size distribution of objects contained in an image. Our method is based on the idea of obtaining a model of the elementary particles that form a texture by optimizing a structuring element to fit the shape of elementary particles. Since the variance of size distribution of a texture is small if the structuring element for calculating the pattern spectrum fits the particles, the shape of elementary particle is obtained by the shape of structuring element optimized to reduce the variance of size distribution. The simulated annealing algorithm is applied to the optimization of structuring element. Our method is also extended to the optimization of gray scale structuring element, which models the particles more precisely. In this case, the optimization algorithm is applied in two stages: The first stage optimizes the extent of structuring element and the second optimizes the pixel value in the extent. Experimental results are shown for both cases of binary and gray scale ones, when the structuring element is restricted to of 5x5-pixels.