The number of workers engaged in Japanese fisheries and related industries in Japan has decreased markedly in recent years due to factors such as an aging workforce, issues related to the management of natural resources and the environment, international affairs, and changes in consumer food preferences. There is therefore a need to mechanize and automate aspects related to work usually performed by humans in the fishery industry. In this research, a system for identifying fish species has been developed. The system employs neural networks which learn to differentiate between different fish species using reference points. Reference points are characteristic points that are extracted from images truss lengths between reference points relative to total body length are used to compile the dataset used for network inputs. For fish with bodies that have been contorted, only data from the vicinity of the fish head are used for network learning. Given that body color is an important characteristic for species identification, an effective method for capturing color data was investigated and the effectiveness of the proposed method and optimal number of color parameters was determined.