I remember asking one of the JTEs (Japanese Teachers of English) that I worked with about the trees and what they were called, and he told me they were「きんもくせい」. He then wrote some characters on the blackboard behind us as told me that these were its kanji: 「金目生」. When I looked it up later, turned out he was wrong. Strange!
Lately I've been thinking about buying one, if I can find the right variety. There are different kinds of sweet olive, and the ones in Japan are aurantiacus. According to my research, a lot of the sweet olive trees from China bloom year-round but aren't as fragrant as the kind that only does so seasonally.
I've found a few vendors online that carry them, but they appear to be the white-flowered, original Chinese version. Hoping I can find the J-variety.
Update: Found it (Nurseries Carolininia)!
Update: Bought a Japanese one and a Chinese one, so I can compare. Also it seems the Japanese variety won't bloom until it grows to about 4', which could take a while, whereas the Chinese kind blossoms from around 6".