We show that a compact self-dual four-manifold with a smooth action of a two-torus and with non-zero Euler characterestic is necessarily diffeomorphic to a connected sum of copies of complex projective planes, and furthermore the self-dual structure is isomorphic to one of those constructed by Joyce in [11]. This settles a conjecture of Joyce [11] affirmatively. Our method of proof is to show, by complex geometric techniques, that the associated twistor space, which is a compact complex threefold with the induced holomorphic action of algebraic two-torus, has a very special structure and is indeed determined by a certain invariant which is eventually identified with the invariant associated with the Joyce's construction of his self-dual manifolds.